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Abstract 

THE EFFECTIVENESS OF 
PARTNERING AND SOURCE SELECTION 
IN JOB ORDER CONTRACTING 

Francis S. Mulcahy 

Chairperson of the Supervisory Committee: 

Professor John E. Schaufelberger 
Department of Construction Management 

Job Order Contracting (JOC) is an alternative to the sealed bid method of procuring 
facility construction and repair services. The concept was developed by the Federal 
Government in the 1980s to reduce the processing time and administrative effort 
required for smaller construction jobs. A typical JOC contract is two to five years in 
duration and work is issued through individual delivery orders, with cost determined by 
pre-negotiated unit prices for a myriad of tasks. However, there are various methods that 
JOC contracts are procured and administered. Due to the need for negotiations on 
individual delivery orders and the lack of a defined scope of work at the beginning of the 
contract, teamwork and frequent communication are required for a successful contract. . 
A formal Partnering process is often used in JOC to focus the parties in that direction. In 
addition, many JOC contracts employ a negotiated source selection procedure to choose 
a qualified contractor instead of using a sealed bid method. 

Research was conducted to examine the effectiveness of Partnering and source selection 
procedures to determine if their use enhances success of JOC. Thirty-five sites 
nationwide were surveyed to assess opinions on performance of the contract from both 
the owner and contractor perspectives. Areas for analysis included construction 
performance, administrative support, the owner-contractor relationship, and the 
participants’ satisfaction with the contract. Analysis of survey data revealed that owners 
and contractors perceived advantages in the use of Partnering and source selection 
procedures in improving key areas of contract performance and overall satisfaction. 
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Introduction 



Job Order Contracting (JOC) is a relatively new method for facility owners to 
contract for construction services. Developed by the Army as an alternative to the 
traditional design-bid-build method of contracting, JOC has been very successful in 
expediting delivery, reducing in-house workload, and allowing owners flexibility, but 
there are problem areas. Awarding JOC contracts based on low bid vice negotiated 
source selection may create situations where negotiated unit prices inaccurately reflect 
the true cost of construction, leading to adversarial relationships and poor performance. 
In addition, the nature of the contract lends itself to a Partnered approach, where all 
stakeholders of a project can trust each other and work together to achieve the best 
results. Non-Partnered JOC contracts may tend toward more adversarial relationships 
and inhibit contract performance. 

This thesis attempts to study the effectiveness of Partnering and source selection 
as they are employed in the administration of Job Order Contracts nationwide. In 
Chapter I, the Literature Review discusses both traditional sealed bid and JOC 
contracting methods in addition to a Partnering and “Best Value” source selection 
procedures. The Research Methodology in Chapter II explains which areas of JOC 
performance were analyzed, how sites were selected for survey, the development of the 
surveys, and the collection of data. The survey response data and the results of the 
statistical analysis are presented in Chapter III, Analysis of Data. Finally, Chapter IV 
contains the conclusions and recommendations of the study. The surveys used in the 
collection of data are included in Appendices A and B, while Appendix C summarizes 
the comments received in the surveys. 
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Chapter I: Literature Review 

1.1 Traditional Sealed Bid Public Works Contracting 

Public works contracts vary from private sector contracts in that legislation sets 
the contracting guidelines for contracts and the public interest demands safeguarding 
process integrity. Thus, the requirement for full and open competition is a hallmark of 
public contracts. To minimize the chance of favoritism or corruption, the sealed bid 
contracting method was developed to award a contract on the basis of price alone. Also 
referred to as the “Design-Bid-Build” process, this system has long been the most 
prevalent option for large and small public works contracts. 

The first step in the procedure is to prepare a design and may be done with in- 
house personnel or through a contract for Architect/Engineer (A/E) services. Generally, 
smaller projects are designed in-house, while A/E designs are used for larger projects 
due to the higher complexity and workload. Once the design is completed and reviewed, 
a bid specification is prepared, then sent out to contractors for preparation of bids. The 
bid specification defines the rules for that particular procurement, including the format of 
the bid, any administrative requirements, and the deadline for bid submission. 

Contractors estimate the cost of the project based on the design and submit a 
lump sum price for a project that remains secret to everyone involved in the process until 
a predetermined time, known as the “bid opening.” At bid opening, the bids are 
compared in a public forum, then the contract is awarded to the bidder who has met all 
requirements of the bid specification, is capable of performing the work, and will charge 
the lowest sum to complete the project. The sum can either be measured as one price for 
the entire project, several prices for different parts of the project, unit prices for elements 
of the project based on estimated quantities, or a combination. 

" 
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The main strengths of sealed bid contracting are its simplicity and objectivity. 
Through the period up to groundbreaking, there is a relatively easy recipe to follow: 
design the project, develop a bid specification, advertise the project, collect and open 
bids, then award the contract to the low bidder. The selection criteria are objective and 
clear; from the pool of bidders who properly and fully respond to the solicitation and are 
deemed responsible to perform the work, the lowest price wins the job. Since no 
judgement is required in the determination of the winning bid, it is difficult to corrupt the 
system with bribery, favoritism, conflicts of interest, or other unseemly methods. 

The most significant benefit achieved through the sealed bid method is the high 
degree of competition that is encouraged. There is no limit to the number of bids that 
may be received in a standard sealed bid procurement, so control of the initial cost of 
construction is excellent. It also allows every responsible contractor in the community 
the opportunity to receive work from the public owner, which is an important social goal 
in most government entities. 

1.2 Issues With Sealed Bid Contracting 



Problems Developed Over Time 

Sealed bid contracts have been used in public works construction for a very long 
time. There have been many developments in both contracts and construction that have 
changed the substance of sealed bid contracting. The technical complexity of projects has 
grown immensely, resulting in the specialization of subcontractors. The lessons of 
construction litigation have been incorporated into contract documents, while litigation 
itself has expanded its role in the resolution of disputes . 1 Contractors have become 
sophisticated experts on contract interpretation in an effort to protect their interests and 



1 Michael T. Kubal, Engineered Quality in Construction: Partnering and TQM (New York: McGraw-Hill, 1994), 
163. 
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maximize profit. 2 Increasing attention has focused on the problem areas within sealed bid 
construction contracting. 



Linear Timeline 

A problem inherent in the sealed bid method is the long lead time required for 
completion of a given project. The cost for a simple “one-follows-another” process is an 
inefficiency in use of time. 3 The private sector, realizing that time is money, has always 
recognized this a major cost and has increasingly abandoned sealed bid contracts. 
Traditionally, public works contracting officials have never placed a high emphasis on 
speed; However, current pressure to make government more efficient has changed this 
paradigm. 4 Though design times will vary based on scope, the procurement process 
times for large and small projects can be similar, which makes the process very inefficient 
for smaller work. 5 a 



Administrative Burden on Small Projects 

The administrative costs of sealed bid procurement must be considered in 
considering the efficiency of the process, especially when compared to the value of the 
construction services acquired. Design contract administration, bid specification 
preparation, advertisement, and contract award expenses may seem relatively minor 
when compared to a $14 million contract for a new administration building, but can be 



2 

Andrew M. Civitello, Contractor \ s Guide to Change Orders (Englewood Cliffs, NJ: Prentice-Hall, 1988), 5. 

^David S. Haviland, Project Delivery Approaches: AnAIA Guide (American Institute of Architects, 1975), 6. 

4 

William B. Moore and Carl F. Stout, Job Order Contracting: A Procurement Success Story (Bethesda, MD: 
Logistics Management Institute, 1988), 2-1. 

5 Dean T.Kashiwagi, Job Order Contracting Performance - 1998 (Arizona State University: Center for Job Order 
Contracting Excellence, 1998), 2. 
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alarmingly large when applied to a $125,000 remodeling project. 6 In most public 
contracting branches, smaller projects significantly outnumber the larger dollar value 
work, which greatly magnifies the inefficiencies’ total effect. 



Adversarial Relationships 

The roles and responsibilities of the various parties have divergent motivations in 
sealed bid contracts. The three-legged stool of owner, designer, and builder is dependent 
on each party fulfilling their separate responsibility to achieve a successful project. 
However, each party’s motivation is to reduce the size of their leg, which has the effect 
of shortening the stool, or in this case, limiting the success of the project. In comparing 
the roles of the parties once design has been completed, it is easy to see how adversarial 
positions are readily generated throughout the project. 7 

The owner’s role in the project, once requirements are identified, is to contribute 
resources in the form of project funding. The sealed bid method ensures that the smallest 
possible initial contribution will be made. Because of the assumption that the design is 
complete and free of defects, the owner views change orders as an attempt by the 
contractor to increase profit and will therefore strive to minimize funding for change 
orders. 8 If the contractor indeed proves that the design is flawed, the owner will look to 
the architect for no-cost redesign or for compensation under design liability if 
construction costs are incurred due to the flaw. Either way, additional funding for design 
flaws will be resisted as well. The result is that the owner will focus on minimizing 
contributions to the project. 

With the design complete, the architect/engineer has already invested the majority 
of effort, and now shifts to the role of design interpreter and advisor during the 



^ashiwagi, 17. 

7 Kubal, 47. 

8 Civitello, 25. 
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construction of the project. However, since the fee for services has already been set, the 
firm will look to minimize the expenditure of additional effort, as that will take away 
from the existing profit . 9 Often, this results in a tendency to interpret design 
inconsistencies in a manner that results in the least additional effort. In admitting a design 
error, the responsibility falls on the architect to provide redesign at no cost. As that 
results in added expenditures, the architect will seek fault elsewhere, as in the owner’s 
miscommunication of intent or the contractor’s poor workmanship. Like the owner, the 
architect’s contributions to project success will always be constrained by a divergent 
motivation . 10 

The contractor in sealed bidding acquires the overwhelming majority of the risk 
during construction, and consequently has the most to lose if the project fails. 

Conversely, the success of the project can be measured in profit earned from the job. 

Since there is a natural conflict between profit and costs throughout the project, the 
contractor’s motivation is to minimize costs, and consequently, his or her contribution to 
the project . 11 The pride in workmanship displayed by the skilled craftsman takes a 
backseat to the fiscal reality of meeting a minimum standard. In some cases, profit as bid 
is minimal even when costs are in line with estimates, and the contractor’s interest in the 
project is minimal as well, diminishing the success of the job . 12 In other cases, the 
contractor’s costs are above estimates and the project is losing money, tempting the 
contractor to reduce costs below the level that would fully satisfy the specifications. 
Ultimately, this divergent motivation has the potential to significantly impact the success 
of the contract. 



9 Kubal, 50. 

^H. Murray Hohns, Preventing and Solving Construction Contract Disputes (New York: Van Nostrand Reinhold, 
1979), 23. 

"Rubai, 74. 

12 Ilohns, 7. 
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The separate responsibilities and conflicting motivations mentioned above create 
a situation that breeds mistrust and conflict between the parties. Clearly, the most 
significant conflict is between the owner and contractor, due to the influence that the 
contractor’s performance has on the outcome, the importance of the outcome to the 
owner, and the proportioning of risk between owner and contractor. 

In many disputes, the owner perceives that the contractor’s goal is to recover 
unanticipated costs, increase profits, or improperly shift risk to the owner. On the other 
hand, the contractor may feel that the owner is trying to get “something for nothing” in a 
change order or improperly shifting risk to the contractor. This adversarial position 
inhibits communication, as each side guards information that would give themselves an 
advantage in negotiation. The hard feelings generated in a dispute will spill over into 
other areas of the work, as resentment leads to poor communication and problems in 
otherwise satisfactory parts of the project. In the end, the attainment of success becomes 
impossible. 

This situation becomes aggravated when the successful low bidder has 
intentionally priced the work at lower than actual cost to “buy in” to the job. The 
contractor starts at a disadvantage, and must either seek to recoup money “left on the 
table” through inflated change order prices or by installing substandard work . 13 



Defective Plans and Specifications Lead to Unpredictable Cost 

The sealed bid system assumes that the contractor can give one lump sum price 
based on a complete set of plans and specifications. If the plans and specifications 
contained every piece of information the contractor needed to perform the work and 
were free of errors, then the price as bid should be the price for the complete project. In 
practice, this rarely occurs . 14 The design package usually contains mistakes, conflicts 

13 Hohns,21. 

14 Hohns, 34. 
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between different parts of the design package, or omissions of necessary details. At best, 
these details are discovered in time for changes to be made at no cost to the owner. At 
worst, work that has already been constructed must be tom out at tremendous cost. 

Most design driven changes, however, will cost the owner extra money and may 
extend the completion date of the project. The sealed bid has been called a “guaranteed 
minimum price,” since the final cost will not be known until after construction is 
completed and all potential contractor claims are resolved . 15 This is especially 
problematic in public works projects, as a fixed budget allowance permits little, if any, 
change in the total contract price. 



Limited Value Engineering 

The sealed bid method only involves the contractor after the design is complete, 
so there is no potential for the contractor to offer valuable advice and assistance in 
development of the design. From methods to materials, the contractor often has a 
realistic account of the constructability, cost elements, and market conditions that will 
affect the project. The designer, though qualified in the art and technology of the 
construction, lacks the contractor’s perspective and makes many assumptions during the 
design process that could easily be clarified through collaboration with the contractor . 16 
Though there is opportunity for value engineering after the contractor reviews the 
design, this is limited by the design as a whole and usually results in little more than 
material substitutions that save only a small percentage of the overall cost. 



1 '’David E. Allen, Changing Delivery Methods: Is Lump Sum Bidding Dead? (Presentation to University of 
Washington CM520 Class, November 18, 1999. 

l6 Haviland, 9. 
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Construction Firms Treated as Commodities 

The major assumption in the sealed bid process, and possibly the most flawed, is 
that all bidders will perform at the same level. Rather than recognizing construction 
management as a professional service, it is viewed as a commodity that can be purchased 
like lumber or concrete . 17 Essentially, performance is treated as a constant and price as a 
variable in the sealed bid equation. 

In reality, each contractor is different, with various levels of qualification, 
experience, training, organization, and competence. Each contractor will approach the 
job differently, with alternative methods, equipment, and schedules. Some contractors 
have developed strong relationships with subcontractors and suppliers that will serve the 
project well, while others have reputations for late payment and unethical practices in 
their dealings with subs and suppliers, setting the stage for work disruptions, material 
delays, and claims. 



Declining Construction Industry Performance 

Though the issues mentioned above have a negative effect on a specific project’s 
success, the impact of many years of sealed bid contracting practices has had an adverse 
impact on the construction industry as a whole . 18 To be a top performing firm in any 
industry, a percentage of the profits must be reinvested in research and development, 
training, and retention of high quality personnel. Sealed bidding forces construction 
profit margins to the lowest possible level, severely curtailing the reinvestment ability of 
the construction industry. Also, there is no reward for outstanding performance in the 
sealed bid system. Superior performance takes extra effort, which drives costs higher, 
making bids higher and less competitive. Performance to the minimum acceptable quality 
level reduces cost and makes bids more competitive. This results in a reverse incentive 

17 Allen. 

IS Kashiwagi, 5. 
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for excellence, a sure sign of an unstable industry. 19 

Dr. Dean Kashiwagi has created a model that shows the relationship between 
competition and performance as it relates to the construction industry (Table 1). 

Table 1: Construction Performance Model 20 









■ 


III 

High Performance 
Perceived High Cost 


II 

High Performance 
Competitive Cost 




IV 


I 

Minimum Quality 
Low Cost 

Declining Performance 



Quadrant I represents the effect of lump-sum low-bid contracting. 
Competition reduces cost, but reliance on low price drives down performance. 

The system has also created an attitude of gamesmanship in many low-bid 
contractors. Those contractors who have been in the sealed bid business for a few years 
have often learned the rules of the sealed bid game “the hard way” after losing money on 
previous contracts. 21 The scenario becomes a “win-Iose” situation, where the owner 
becomes the opponent and “tactics” are employed to maximize profit. Although there is 
high competition for the initial bid, contractors recognize that once they are awarded the 



19 Kashiwagi, 5-7. 
20 Kashiwagi, 6. 

2 'civitello, 5. 
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job, they are in a sole source negotiating arena concerning change orders . 22 Instead of 
waiting for the discovery of change orders, contractors “prospect” for errors, omissions, 
or conflicts in the design . 23 After assessing the potential value of changes, strategies for 
resolving the changes are developed to maximize the contractor’s advantage during 
negotiation. 

1.3 Job Order Contracting 

Background 

Job Order Contracting (JOC) was developed by the Army in Europe as an 
alternative to the traditional sealed bid method of construction contracting . 24 At the time, 
there was tremendous pressure on the Army contracting officials to reduce the timeframe 
needed to procure construction services, to make progress on a backlog of work orders 
despite a shrinking in-house staff, and to improve quality. JOC was conceived as a way 
of rolling many small jobs into one large contract awarded to a single provider using a 
unit price format. The main goal was to minimize the administrative contracting burden 
by eliminating the need to develop separate bid specifications and competitively bid each 
single job. The large contract’s general requirements could be negotiated once, scaling 
back managerial requirements to just scope definition and quality issues. In addition, it 
was hoped that dealing with one contractor would significantly reduce the in-house 
effort to supervise a myriad of small contracts with many different contractors. 



22 Hohns, 9. 

23 Civitello, 87. 

24 

Leif T. Erickson and Patrick D. Murphy, “The Job Order Contracting Solution”, Civil Engineering (April 1994), 
70. 
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The JOC Concept 

A Job Order Contract (JOC) is an indefinite delivery-indefinite quantity contract 
that usually is awarded for one year with up to four option years. Specific projects are 
not defined at time of award. Instead, the contract encompasses a broad selection of 
facilities construction, maintenance, repair, and renovation type work that may develop 
during the duration of the contract. Unit prices determine the overall project cost, but 
JOCs vary from traditional unit price contracts in that the basis for pricing is a pre- 
published unit price book that covers costs for thousands of individual construction 
tasks. The price book is part of the contract specifications and can be either a 
government-produced book or a private sector estimating guide, such as the Means 
Estimating System. 25 The contractor’s bid is actually a coefficient applied to the book’s 
unit prices to decide the price of work. The coefficient covers the contract’s general 
requirements, taxes, labor burden, overhead, and profit. For example, a coefficient bid of 
1 .06 would increase the as-published cost of line items by 6% to cover all costs, 
overhead, and profit of the work, with no other markup required. For example, a line 
item for installing nylon carpet with a list cost of $20.84 per square yard would be priced 
at $22.09 per square yard under this contract. 

The process for issuing work under the contract is relatively simple, as the 
contractor has already been selected and has satisfied all of the general requirements such 
as bonding, safety plans, quality control plans, and insurance. The level of design may 
vary greatly from a site visit with the requesting party to verbally communicate the work 
requirement to a fully designed package that was prepared for sealed bid. Once the 
contractor and owner agree on a scope of work, a detailed cost estimate is submitted by 
the contractor listing all unit prices required for the job. After the owner verifies that the 
line items and quantities listed are proper, then a job order is issued and construction 



'^Erickson, 69. 



) 



begins. The process from identification to construction ranges between 20 - 30 days. 26 
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Advantages of JOC 

The JOC concept has several key advantages over the traditional design-bid-build 
method that have impressed facility owners in both the public and private sectors. 

Though the most significant benefits are attained through accomplishment of small repair 
and renovation jobs, the advantages apply to large JOC construction projects as well. 



Response Time 

The most recognized benefit of JOC is by no coincidence the reason it was 
created in the first place: to shorten the response time for customers requesting 
construction services. Since the JOC contractor has already been selected, the time 
required for bid specification development, advertisement, solicitation, and contractor 
award are eliminated for individual projects. 27 Since the JOC process does not require a 
full design, a site visit is all that is needed to convey information to the contractor. For 
very small jobs that are simple in scope, this usually means that a design can be bypassed 
completely, with the contractor providing details on how the work as scoped should be 
accomplished. The owner, rather than creating the design, simply reviews and either 
modifies or approves the proposal. 28 A job order can be issued to the contractor in less 
than 30 days of a customer request, with a comparable project taking up to 180 days to 
award a contract under the design-bid-build process. 29 A comparison of Army 
procurement times between JOC and non-JOC projects showed that JOC delivery orders 



26 Kashiwagi, 19. 
27 Kashiwagi, 20. 
28 Erickson, 69. 
29 Kashiwagi, 19. 
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are issued in 18-27% of the time it takes for non-JOC awards. 30 



Reduced Administrative Burden 

With the elimination of boilerplate requirements from individual jobs, the 
administrative burden of JOC construction is minuscule compared with design-bid-build. 
Paperwork is reduced to only those documents pertaining to the technical aspects of the 
work. The information generated during site visits, scope definitions, discussions on cost 
estimates, and schedule delineation is recorded in a streamlined manner and contained in 
a file for later reference. Though quality assurance and supervision are still required, the 
ease of working with one contractor as compared to interfacing with small individual 
contracts is invaluable. For instance, the procedures for accomplishing work at a 
particular site need only be explained once, unlike separate bids where individual 
preconstruction meetings and training are required for each job. In addition, the owner 
and contractor personnel that interface regularly become familiar with each others 
processes, minimizing confusion and promoting efficiency. Because the contracting staff 
spends less time on individual procurements, they are freed up to process a higher 
number of work requests, which allows for a rapid reduction in maintenance backlog. 31 



Lower Design Costs 

As stated above, design costs can be practically eliminated for smaller work 
items. Though an owner-provided design may be necessary for larger and more complex 
JOC projects, design level of effort may be reduced to provide only those details that are 
outside the contractor’s area of expertise. This “shared” design effort means that for a 
building electrical upgrade, the designer would focus on the building’s load and required 



30 Moore, 2-2. 

3 1 Moore, 2-11. 
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panel and transformer size, while the contractor could determine conduit routes and 
receptacle placement. Since most JOC contract specifications include a requirement that 
all work meets applicable codes, many typical bid specifications and submittals can be 
omitted. In this maimer, effort of the Architect/Engineer is reduced and design costs are 
minimized . 32 



Value Engineering and Constructability 

In those cases where design is required, the JOC contractor can participate in the 
preparation of the design and advising the designer on constructability issues. In the 
electrical upgrade example cited above, the contractor may not only reduce the design 
effort, but reduce the construction cost as well. By teaming with the designer to optimize 
placement of the transformer and electrical distribution panels, the amount of structural 
demolition and length of cable runs can be reduced. In a traditional design, the designer 
applies his strength, engineering knowledge, along with a weakness in constructability 
experience. The inclusion of the contractor as part of the design team eliminates the 
weakness, creating a much more complete and cost-effective design . 33 This also allows 
for a shorter construction time, as the contractor is intimately familiar with the design 
and can influence the design to allow for speed of construction. 



Teamwork 

The traditional sealed method promotes adversarial relationships, yet the JOC 
format lends itself to an environment of teamwork and cooperation. The long-term 
nature of a JOC encourages both contractor and owner personnel to work together to 
achieve a mutually beneficial relationship. 



32 Erickson, 69. 
33 Erickson, 69. 
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For the contractor, the incentive is future profit. Since the price is already set by 
the coefficient, the contractor can only increase profit in an individual job order by trying 
to negotiate unreasonable line items and quantities or by reducing quality of 
workmanship. If negotiations become difficult and quality suffers, the owner will choose 
another contracting avenue and few job orders will be issued to that contractor. 34 On the 
other hand, if negotiations are straightforward and honest, and high quality workmanship 
is the norm, then the JOC contractor can expect many more delivery orders. 35 The 
mathematics are simple: small profit on many projects is greater than a large profit on a 
handful of jobs. The natural incentive, then, is for good performance at reasonable prices 
and a pleasant experience for the owner. 

The owner’s representatives also have a vested interest in the success of the 
contract. For those sites with unpleasant experiences in sealed bid contracts, a large 
backlog of work requests, and impatient customers, a JOC can be an exceptional way to 
accomplish work compared with other methods available. In fact, 1 00% of Army 
Installation Commanders surveyed in 1988 declared JOC a “must-have” contracting 
tool. 36 Like the parable of “The Goose that Laid the Golden Egg,” the last thing the 
owner wants to do is kill the goose. Though not abandoning the requirement to 
safeguard their interest, the understanding that an adversarial relationship can eliminate a 
viable contracting option encourages owners to treat the contractor with the respect due 
a valued business partner. Sometimes, this can be a difficult transition for those who are 
ingrained with the “cops and robbers” mentality of sealed bid contracts, but many 
officials welcome the chance to relinquish hard line stances and conduct business in an 
environment of respect and open communication. 



34 Erickson, 70. 
35 Kashiwagi, 20. 
36 Moore, 2-4. 
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Enhanced Quality 

Free from focusing on administrative requirements and adversarial positions, the 
owner and contractor personnel can turn to the important elements of construction: time, 
quality, and customer satisfaction. The teamwork that exists between contractor and 
owner allows the tenant to be intimately involved in all phases of the job, from design 
requirements to scheduling outages and work disruptions . 37 The job can be scheduled 
better since tenant and contractor needs can be meshed through open discussion, and 
contractor work stoppage can be minimized, keeping costs down. The natural incentive 
for a quality product and a satisfied customer, if fostered by all parties, will result in a 
successful project . 38 



Minimal Change Order Rate 

JOCs typically enjoy a very low change order rate due to the integration of the 
contractor in scope definition and design, the pre-negotiated unit price system, and the 
shift away from adversarial relationships . 39 The changes that do occur are mainly 
customer requested, due to the pre-priced nature of modifications and ease of execution. 
The joint scoping process and open communication during construction generates a 
valuable flow of ideas that obviate changes, resulting in a reliable up-front cost. 

Because the claims process is inherently adversarial, its use is avoided in JOC. 
Both contractor and owner work to resolve any differences at a lower level, realizing 
that a claim on one delivery order can inhibit the performance and/or award of future 
delivery orders. 



J7 Moore, 2-6. 
38 Moore, 2-6. 
39 Erickson, 70. 
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1.4 Partnering 
The Partnering Concept 

The inherently adversarial positions in sealed bid contracting, with each side 
maintaining divergent motivations and general mistrust of the other, place barriers 
between the parties when a disagreement arises. The atmosphere of mistrust, combined 
with the litigious bias of modem construction disputes, results in carefully screened 
communication and a “win-lose” mentality . 40 True resolution of issues cannot occur in 
such an environment. Formal Partnering seeks to avoid disputes through frequent and 
open communication based on a relationship of trust . 41 

Partnering was developed to prevent barriers from springing up in public works 
projects by keeping the parties committed to a common vision and fostering trust among 
participants. The common vision is determined by a consensus of the group, and usually 
focuses on goals for project success, including customer satisfaction, quality, time, and 
dispute resolution. Trust leads to frequent and open dialogue, which is essential in 
resolving disputes. It is through participants working together to achieve a common 
vision, with a foundation built on strong communications and trust, that successful 
projects are realized . 42 

The successes of Partnering are well documented. In recent years, the use of 
Partnering on construction projects has seen a dramatic increase . 43 Its cornerstone is a 
partnering agreement where both parties focus on a common set of project objectives 



"^Thomas R. Wame, Partnering for Success (New York, NY: ASCE Press, 1994), I. 
41 Wame, 5. 

42 Kubal, 106. 

43 Wame, 70. 
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and problem solving methods. 44 It is best applied in large, long-term projects where the 
relationship can be developed and there is an incentive to work together. 45 The Army 
Corps of Engineers pioneered use of Partnering in the 1980s and has seen a dramatic 
decrease in its volume of contract claims and appeals. From 1988 to 1994, claims were 
down 71%, while 365 appeals were filed in 1994, down from 742 appeals in 1991. 46 The 
Air Force used Partnering on a $226 million Large Rocket Test Facility, finishing 1 14 
days ahead of schedule and $12 million under budget without any claims or appeals. 47 
The Navy has implemented Partnering on its large Base Operating Support Contracts 
and Job Order Contracts, as well as some individual projects. 



Implementation of Partnering in JOC 

The format of the JOC contract encourages teamwork through the long-term 
relationship and inherent incentives that exist for each stakeholder. 48 The implementation 
of a Partnering program formalizes these incentives into a tangible agreement and is a 
natural extension of the JOC concept. The first step in the process is to agree to Partner, 
which should be done as soon as possible after contract award. Each party should select 
a Partnering “champion,” a member of management who will be on a site for the 
duration of the contract and will ensure that the Partnering environment is promoted and 
maintained. 49 



M Wame. 13. 

45 Kubal, 129. 

46 

Administrative Conference of the United States (ACUS), Toward Improved Agency Dispute Resolution: 
Implementing the Administrative Dispute Resolution Act (February 1995) Ch 3, Sec II. 
<www.adr.af.mil/afadr/libraiy/docs/acusrpt.html> 

47 

U.S. Air Force, Memorandum Advances in Alternative Dispute Resolution (ADR). 
<www.adr.af.mil/afadr/library/docs/secrep2.html> 

48 Kashiwagi, 20. 

49 Wame, 7. 
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An initial Partnering session should then be scheduled before commencement of 
work under the contract. During this session, which may last one or more days, the 
stakeholders become familiar with one another and the overall objectives of Partnering. 
The next step is to develop a charter that will outline the mutual commitment to success, 
how that success will be measured, what tasks must be completed to achieve success, 
and a methodology of how to resolve disputes on the way to success. Finally, the parties 
must carry this charter with them into the project and treat it as a living document. 
Successful Partnering takes a daily commitment to its principle and the objectives set 
forth in the contract. Follow-on sessions should be scheduled to perform “routine 
maintenance” and to ensure that the charter is being adhered to . 50 

1.5 “Best Value” Source Selection Procedures 

The Best Value Concept 

“Best Value” Source Selection is a term coined in the Federal government for a 
contractor selection process developed as an alternative to the sealed bid method. It is 
based on the common sense premise that price alone may not be an acceptable criterion 
in purchasing products or services. Rather than selecting the lowest price, an owner 
applying source selection procedures determines the criteria that is important for the 
project, and invites contractors to submit proposals that will be measured against that 
criteria for final selection. The process allows for negotiations between the owner and 
interested parties, so that all necessary information may be gathered before a selection is 
made. 

The criteria vary for each procurement, but price is always considered to some 
degree. Common evaluation factors include past performance, key personnel 



,0 Wame, 36. 
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qualifications, operational methods, and organization plans. Project specific criteria, such 
as schedule, value engineering proposals, or design considerations are also standard. 
Since the relative importance of a given factor will vary between projects, each criterion 
is assigned a weighted value in the source selection plan . 51 

After negotiations have been conducted, the contractor proposals are reviewed 
and graded by an evaluation board consisting of a group of officials specifically 
appointed for the project. The proposals are then ranked based on their weighted grades, 
then compared with their price to determine which proposal represents the best overall 
value to the owner. Typically, the owner is willing to award the contract to other than 
the low-priced offerer since a modest increase in price may gamer a higher quality 
product with lower risk of nonperformance or failure. 



Source Selection Advantages and Disadvantages 

The biggest advantage to source selection is the ability to apply business 
judgement to procurement. By selecting criteria that are key to the procurement and 
choosing a contractor based on the criteria, the owner is “steering” the procurement in a 
positive direction, vice the random chance element of sealed bid . 52 In avoiding the sealed 
bid practice of buying into the job, the source selection process allows the owner to pay 
the “right price” initially, negating the need for the contractor to make profit up through 
later modifications . 53 This reduces adversarial tensions and makes for smoother contract 
administration, with the added benefit of more predictable costs. By examining the 
contractor’s past performance and qualifications of key personnel, those contractors that 
have the strongest resume and present the least risk of poor performance would stand a 



5 * “Best Value Buys.” Government Executive Magazine (online), January 1995. 
<www.govexec.com/procure/artides/0195prs6.htm> 

52 “Best Value Buys”. 

53 Sherie Winston, “Pentagon Pumps Up Performance.” Engineering News Review (online) (October 1999), 
<www.en r.com/n ews/en rb 1 1 OO.asp> 
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higher chance of selection . 54 

The negotiation process also promotes communication between the owner and 
potential contractors, which leads to clarification of scope and technical assumptions 
prior to contract award . 55 Unlike sealed bidding, the proposal can explain the price by 
listing the assumptions that the estimate was based on. If contingencies were included 
due to uncertainties in the project, they can be clarified by the owner during negotiations 
with a reduction in cost. Likewise, if the cost as proposed left out necessary items, the 
contractor can add those in the initial price, saving a potential change order down the 
line. 

The key disadvantage is the additional administrative effort required to conduct 
the source selection process compared with the simple sealed bid process. Though 
selection procedures may be streamlined to fit different scale procurements, they still 
consume valuable resources like time and personnel. Also, some question why the public 
owner should pay anything above the lowest price offered, and because the use of 
subjective judgement is involved, bid protests and process integrity is of concern . 56 

1.6 Summary 



The Job Order Contracting format promises the facility owner a cost-effective 
way to expedite the accomplishment of facility construction and repair projects with a 
higher quality and reduced administrative effort. Though the requirement for a design 
may vary by project, the contractor should always be integrated into design development 
as early as possible to ensure the best result. 



54 Winston. 
55 Winston. 
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Selection of a JOC contractor should be one through a source selection 
procedure that emphasizes the best value to the owner vice the lowest price. Because of 
the long-term nature of the relationship, it is important to apply business judgement in 
the choice of an important partnership and the pitfalls of sealed bid contracting should be 
avoided. Once selected, the partnership should be formalized through a Partnering 
agreement to define the common vision and guiding principle of the participants. 

By using the JOC format, selection of a quality contractor, and committing to a 
shared vision based on open communication and trust,, public facility owners can 
revolutionize the way they accomplish construction and repair projects, increasing 
customer satisfaction and reducing staff workload in the process. 
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Chapter II: Research Methodology 



To study the effectiveness of Partnering and source selection in the 
administration of JOC, a survey method was chosen to evaluate the opinions of both 
owner and contractor project managers on the performance of their contract. The survey 
was designed to measure a broad range of issues, from direct construction of the project 
to the elements forming the contractual relationship between the parties. 

The survey responses would be segregated into four different populations for 
comparison between each other: 

1 . Partnered contracts procured through a negotiated source selection using a Request 

For Proposals (RFP) 

2. Non-Partnered contracts procured through a negotiated source selection a RFP 

3. Partnered contracts procured through competitive bidding (low-bid) 

4. Non-Partnered contracts procured through competitive bidding (low-bid) 

The anticipated results should show that Partnered contracts where the source 
selection process is used have the highest performance levels of all four populations, 
while non-Partnered/Low Bid contracts should have the lowest levels of performance. 
The other two sample populations’ performance levels should fall somewhere in 
between. 



Types of Contracts Studied 

Job Order Contracting had its start with the U.S. Army, but the concept quickly 
spread to the other Armed Services, then throughout the Federal Government. Today 
many state universities employ the JOC concept as well. However, though the concept 
remained the same, the names that were given to the contract format changed. The Army 
and Navy still refer to the concept as JOC, but the Air Force has termed it “Simplified 



Acquisition of Base Engineer Requirements” or SABER. The Coast Guard calls it an 
IDIQ contract, for Indefinite Delivery, Indefinite Quantity. The various universities that 
were studied had various names for the JOC process, but the concept remained intact. 

Therefore, the study was limited to JOC, SABER, Coast Guard IDIQ, and 
simplified delivery order type contracts that met the following criteria: 

1 . Work issued through individual delivery orders under an umbrella contract 

2. Line item pricing based on a unit price book with contractor-bid coefficient 

3. Multi-trade discipline capability 

4. Potential long term contractual relationship (2-5 years) 

Contracts eligible for study included only those that were ongoing or were in the 
closing stages of completion to ensure project managers were very familiar with 
performance. Contracts that had not been in place long enough for a track record of 
performance were not included. 



Areas for Analysis 

Based on the research conducted in the literature review, four key elements of 
the JOC process were identified as important topics for data collection. These elements 
consist of the performance of construction under delivery orders, the administrative 
support required for construction, the quality of relationship between the parties, and the 
overall level of satisfaction with the contract. 

The performance of actual construction manifests itself in much the same manner 
as traditional construction. The three basic concerns for a construction project that 
would be measured in the survey are quality, on-time completion, and safety. In addition, 
job order contracting is heavily reliant on subcontracting, so the scheduling and 
performance of subcontractors are of major importance for a successful JOC contract. 
Also, warranty service is an area of concern with so many small projects issued under the 






26 



contract and needs to be addressed. 

The JOC concept is mainly celebrated for its streamlined nature and high level of 
responsiveness to an owner’s needs. Since many projects issued under a JOC are of an 
urgent nature and must be accomplished in the most efficient manner possible, the 
success of this type of contract heavily relies upon the support element. Therefore, 
information needs to be gathered on all areas of administrative support for JOC 
construction. Much of this support must work both ways for a contract to be successful, 
so most of the data was gathered from both the owner and contractor viewpoints. Key 
areas of support included: 

* Responsiveness and timeliness of administrative support (requests-for-information 

(RFI), submittals, reports, etc.) 

* Innovation and Value Engineering (contractor’s generation of, owner’s reception to) 

* Contractor’s responsiveness to special needs of the clients 

* Ability to solve and prevent scheduling and site coordination problems 

* Contractor’s management effectiveness (ability to handle many simultaneous orders) 

* Level of administrative effort required of both owner and contractor 

* Reasonableness of the owner’s inspection or Quality Assurance program 

The foundation of success in both the construction and support elements is the 
underlying relationship that exists between the contractor, owner, and in most cases, the 
end-user. Partnering was designed to apply a formal structure to this foundation and 
strengthen the connections that are necessary for a successful project. Since the essence 
of Partnering is a combination of trust and open communication, the levels of trust and 
communication were measured from both the contractor and owner perspectives. The 
quality of the relationship can also be measured in the ease of the many repetitive 
delivery order negotiations and in the resolution of changes, claims, and disputes as they 
arise throughout the construction process. 
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Finally, the overall satisfaction of the parties with the contract is the fourth key 
element and is the culmination of the parties’ effort in the first three elements. In the end, 
it is the most important, not only because it measures the combined success of 
construction, support, and relationships, but mainly because it determines the actions of 
the parties after the project is complete. A satisfied owner will likely expand the concept 
further, leading to better service and streamlined administration, while rewarding the 
good performance of the contractor with solid references or even award of more 
construction. The overall satisfaction of the parties is measured in three areas: 
satisfaction of the owner and contractor, satisfaction of the end-users of the 
construction, and the owner’s perception of efficiency as compared with other contract 
vehicles. 



Selection of Projects for Survey 

The search for potential sites was undertaken to collect as many data points as 
possible for the analysis. Established firms that specialize in JOC were contacted for 
potential sites. The Center for Job Order Contracting Excellence at Arizona State 
University also provided some potential sites for analysis. The Army and Navy provided 
points of contact. In addition, as sites were contacted to participate in the survey, new 
leads were generated. In the end, 35 sites met criteria as candidates for study. The sites 
are broken out as follows: 



U.S. Army/Corps of Engineers 


14 sites 


State Universities 


7 sites 


U.S. Air Force 


6 sites 


U.S. Navy 


5 sites 


Other Federal Government 


3 sites 
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Survey Development 

Two questionnaires were developed to measure contract performance at each 
site. One was designed for the owner’s JOC project manager and the other was designed 
for the contractor’s project manager at the local site level. The evaluation consisted of 
two basic scoring systems, a numeric score (from 1 to 5, where 1 = Poor, 5 = 
Outstanding) and a multiple choice descriptive answer. The Owner and Contractor 
questionnaires are included as Appendices A and B, respectively. 

Several questions are duplicated in both questionnaires, as they measure areas 
common to both parties such as levels of trust and communication. Other questions were 
designed for the respondent to grade the performance of another party, such as the 
owner’s scoring of the contractor’s management effectiveness. In addition, respondents 
are asked to make comments about the JOC process at the end of the questionnaire. 

The owner’s questionnaire consists of twelve numeric scores and nine descriptive 
answers for a total of 21 questions. The contractor’s version contained three numeric 
scores and six descriptive answers for nine questions. Questions 13 and 14 of the 
owner’s questionnaire were not assessed for this study, as they were included to gather 
data for a separate research paper. 

The JOC/SABER Contractor Performance Questionnaire developed by Arizona 
State University’s Center for Job Order Contracting Excellence was used as a model for 
the development of these questionnaires. 



Data Collection 

70 questionnaires were mailed or faxed to respondents at the 35 sites from April 
14, 2000 to May 1, 2000. 63 questionnaires were returned by the cutoff date of May 19, 
2000, for a 90% response rate. After minor clarification on some survey data, the 
questionnaires were then segregated into the four population groups shown in Table 2. 
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Table 2: Sample Population Summary 



# of Sites 


RFP Source Selection 


Low bid 


Partnered 


17 (49%) 


1 (3%) 


j Non-Partnered 


10 (28%) 


7 (20%) 



Of the 35 sites surveyed, both owner and contractor questionnaires were received 
for 28 sites, while at least one questionnaire was received for the remaining seven sites. 
As only one site was classified as a Partnered/Low bid site, there was insufficient data to 
analyze for that population and it was excluded from study. 
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Chapter III: Analysis of Data 



The data collected from the survey responses was divided into three sample 
populations for analysis. Note that sample sizes for the two Non-Partnered groups are 
somewhat small and will be a concern during the statistical analysis. A summary of site 
totals and survey responses is depicted in Table 3. 

Table 3: Survey Response Summary 



Sample Population 


Number 
of Sites 


Owner 

Questionnaires 

Received 


Contractor 

Questionnaires 

Received 


Overall 

Response 

Rate 


Partnered + 


17 


16 


16 


94% 


RFP Source Selection 










Non-Partnered + 


10 


8 


9 


85% 


RFP Source Selection 










Non-Partnered + 


7 


6 


6 


86% 


Low Bid 










Totals 


34 


30 


31 


90% 




Survey response data was tabulated by sample population and by question. A 
summary chart displays the responses based on the two answer formats, numeric and 
descriptive. For numeric scores, the means for the three sample populations are 
presented side by side. For those questions where both contractor and owner supplied 
answers, both sets of data are presented on one chart. For descriptive answers, the 
number of responses for each answer is presented for all three sample populations 
studied. Due to the complexity of the chart type, only contractor or owner answers will 
be shown on each chart. 
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The following chart abbreviations will denote sample populations: 

P-RFP: Partnered contracts procured via Request For Proposal (RFP) negotiated 

source selection 

NP-RFP: Non-Partnered contracts procured via RFP negotiated source selection 

NP-LB: Non-Partnered contracts procured through competitive bidding (low-bid) 

The following chart abbreviations will denote descriptive answers: 



VH: 


Very high 


MB: 


Much better than 


VR: 


Very reasonable 


H: 


Higher than normal 


SB: 


Slightly better than 


R: 


Reasonable 


A: 


Average 


S: 


Same as 


SU: 


Somewhat 












unreasonable 


L: 


Lower than normal 


SW: 


Slightly worse than 


U: 


Unreasonable 


VL: 


Very Low 


MW: 


Much worse than 


EXC: 


Exceptional 



3.1 Performance of JOC Construction 

The performance of construction is analyzed to determine if a difference in 
performance is apparent by varying the procurement and administration methods. 
Owners were asked to grade the contractors’ efforts in quality, safety, on-time 
completion, subcontractor scheduling/performance, and warranty service. All answers in 
this element are numeric scores. Ideally, the combination of Partnering and RFP Source 
Selection should result in higher scores across the board in this element. 
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Quality of Construction 

According to the responses shown in Figure 1 , it would appear that the RFP 
process does not affect quality but that Partnering has a tremendous effect on 
construction quality. This is somewhat surprising, since the goal of the RFP process is to 
ensure a higher quality product than the low bid process provides. This could indicate 
that the RFP process is ineffective or that Partnering has a much larger effect on quality 
than the procurement process. 
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Quality of Construction 

Summary of Sample Means 




P-RFP NP-RFP NP-LB 

Sample Population 



Figure 1: Quality of Construction Summary 



Comments from respondents generally did not address quality. Among 
Partnered/RFP projects, the few comments made were positive. One Non-Partnered/RFP 
owner specifically addressed a lack of sufficient quality control that led to rejection and 
rework of installed material. One Non-Partnered/Low bid contractor was praised for 
outstanding performance. 
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Contractor’s Safety Performance 

Data on contractor safety in Figure 2 actually shows that safety performance is 
slightly higher under low bid than RFP given a Non-Partnered environment. However, 
the Partnered/RFP environment seems to produce the safest job sites. This may be 
attributed to the increased level of communication on Partnered contracts as well as a 
common goal of an accident-free jobsite that many Partnering charters contain. 
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Figure 2: Safety Summary 



On-Time Completion of Job Orders 

The distribution of responses in Figure 3 fits the expected pattern in this area. 
RFP contracts perform slightly better than low bid contracts in a Non-Partnered 
environment, but the addition of Partnering results in a much higher performance score. 
This may be attributed to the tendency of Partnering to resolve issues disputes quickly at 
the lowest levels before a dispute can cause costly delays to the work in question. The 
increased communication at the beginning of the project may also prevent modifications 
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that delay the project. 



On-Time Completion 

Summary of Sample Means 
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Sample Population 



Figure 3: On-Time Completion Summary 




Comments were not directed at completion time, but a couple of comments from 
Non-Partnered/Low Bid sites indicated that the process is slowed if the government has 
tight restrictions over design. 
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Subcontractor Scheduling and Performance 

The results in Figure 4 indicate a much higher performance level in a 
Partnered/RFP environment than in the other two formats. Traditionally, JOC 
contractors staff only the construction management personnel in-house, subcontracting 
for trades in the execution of delivery orders. This is mainly due to the uncertainty of the 
type of work issued under a JOC. Usually, the JOC contractors will try to develop 
relationships with the better subcontractors in the area, as better work can be expected 
with less hassle and risk than using less qualified subs. In the JOC business, better work 
leads to more work. If a low coefficient does not allow use of better contractors, 
problems may result. 

Scheduling/Performance of Subs 

Summary of Sample Means 
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Figure 4: Scheduling/Performance of Subcontractors Summary 

One contractor in a Partnered/RFP environment cites “working with 
subcontractors that we haven’t been comfortable with” as a reason for a slow start to a 
contract. A Non-Partnered/RFP owner blamed “a lot of unqualified subcontractors” for 
significant rework. A Non-Partnered/Low Bid owner drew a direct conclusion between 
an artificially low coefficient, unproven subcontractors, and user dissatisfaction. 
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Warranty Service 



Warranty Service 

Summary of Sample Means 

-4:43- 




P-RFP 



NP-RFP 

Sample Population 



NP-LB 



Figure 5: Warranty Service Summary 



The trend in Figure 5 is expected, as source selection seems to add improvement 
over low bid, then Partnering results in even higher performance. Basically, the key here 
is money. Often, a low coefficient allows for enough money to complete the work, but 
little or no contingency for rework. Since some warranty issues are a grey area at best, it 
can become quite contentious when nobody owns up to responsibility for fixing an item 
under warranty. Quite possibly, the Partnering environment may lead to quicker 
resolution of warranty issues and therefore higher performance. 



3.2 Performance of Construction Support 

The contractual support that the parties in a construction project provide each 
other with is critical in achieving a successful outcome. Decision-making forces rely on 
the information and support provided to keep the project on schedule and within budget. 
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The true value of the management of a JOC on both sides of the contract rests with the 
ability to get ahead of the project and stay ahead of it though its duration. The following 
areas will measure the performance in this critical area. 



Responsiveness and Timeliness of Administrative Support 

This area was intended to measure the responsiveness of the owner and the 
contractor in providing support to each other. The owner was asked to grade the 
contractor’s responsiveness and timeliness with cost estimates and drawings, while the 
contractor was asked to grade the owner’s responsiveness with support such as requests- 
for-information (RFI) and submittals. 



Responsiveness of Support 

Summary of Sample Means 




Sample Population 

Figure 6: Responsiveness of Support Summary 

The owner responses in Figure 6 fit the general expected pattern. However, the 
contractor answers vary from the expected trend. In theory, the trends of the owner and 
the contractor should be somewhat similar. In rereading both survey questions, I realized 
that the owner’s questionnaire was straightforward in nature, while I noticed that the 
contractor question could be inferring that the contractors should grade themselves on 
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the support they provide to the owner in terms of RFIs and submittals. I suspect that the 
vague wording of the question could have led to the irregular data pattern. 

Innovation and Value Engineering 



Figure 7: Innovation and Value Engineering Summary 

The owner was asked to grade the contractor on the quality and frequency of 
innovative ideas and Value Engineering proposals, while the contractor was asked to 
grade the owner on the reception to innovation and Value Engineering proposals. The 
intent was to measure the climate for innovation among the various formats. There is a 
consistent connection between the contractor and owner answers here, as expected, so 
the data looks reasonable. Based on the data in Figure 7, there appears to be a healthy 
climate for innovation in the Partnered format that the other formats do not seem to 
share. The Partnered owner and contractor scores are also much closer together, 
indicating more of a consensus view. 



Innovation and Value Engineering 

Summary of Sample Means 




Sample Population 
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Responsiveness to Client Needs 

This question is measuring the ability of the contractor to respond to the needs of 
the clients using the JOC. The Partnered contracts in Figure 8 show much higher scores 
than the other two formats, which may be attributed to the greater involvement of the 
end-user in determining the scope of the work and greater communication among the 
project stakeholders. 



Responsiveness to Client Needs 

Summary of Sample Means 




P-RFP NP-RFP NP-LB 
Sample Population 

Figure 8: Responsiveness to Client Needs Summary 



As for the drop in the Non-Partnered/RFP format score, this can be attributed to 
two sites that had problems with this very area. The first owner commented that the 
contractor came in with a limited staff into an environment where they were in direct 
competition with in-house forces. The feeling was that they should be doing more to 
please the clients in order to gamer more work. The second owner was an Air Force 
owner working through an Army agency who sees “a large communications gap from 
the Air Force’s request to the contractor’s delivery.” This could be due to the structure 
of the contract organization vice the procurement method of the contract. 
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Ability to Prevent and Solve Scheduling and Site Coordination Problems 

The owner was asked to evaluate the ability of the contractor to prevent and 
solve scheduling and site coordination problems. This is a key area of concern for a JOC 
since many projects involve repair or renovation of an existing facility and are of an 
urgent nature where construction work takes place in shorter time frames. This summary 
(Figure 9) shows that there is a much higher score for Partnered projects and very little 
difference between the Non-Partnered formats. 

Preventing & Solving Problems 

Summary of Sample Means 

4.16 




Sample Population 

Figure 9: Preventing and Solving Problems Summary 

Though not fitting the expected pattern for Non-Partnered/RFP, the Partnered 
scores were expected much higher as this is one of the main strengths of Partnering. 
Many problems with construction site coordination and scheduling can be traced to 
miscommunication or non-communication. Also, a lack of mutual trust leads the parties 
to suspect one another when a problem creeps up and may inhibit communication. With 
Partnering, scheduling and coordination problems are more likely to be aired quickly and 
resolved due to the open nature of the relationship. An owner who specifically identified 
a communication problem in a JOC contract also stated “It seems that scheduling 
problems, site coordination problems, and dissatisfaction is the norm.” 
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Contractor’s Management Effectiveness 

The management effectiveness was evaluated due to its significance in JOC. The 
contract can be difficult to manage since it is made up of many small delivery orders 
comprising many different types of work. Often, there are dozens of jobs going on 
simultaneously in widespread locations throughout a site. On top of that, the urgency of 
many JOC jobs requires expedited materials, flexible scheduling requirements, and 
working around tenants. The owner’s perception of the contractors management 
effectiveness is a key indicator of a successful project. The Partnered sample population 
in Figure 10 has much higher test scores in this field, with the Non-Partnered scores 
essentially the same. 
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Management Effectiveness 

Summary of Sample Means 
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Figure 10: Management Effectiveness Summary 



Level of Administrative Effort 

In theory, the owner’s administrative effort should be reduced in a Partnered 
environment, since the need for dispute-oriented documentation is greatly eliminated. 
Though there is an additional effort required for Partnering, it is seen as an up-front 
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investment that should save time later in the project by preventing problems and quickly 
resolving those that do come up. Also, RFP projects should have reduced administration 
costs due to the selection of a qualified contractor and presumably an adequate 
coefficient. 

Analysis of the responses, shown in Figure 11, indicates that over half of the 
owners on negotiated contracts assessed the administrative effort as average or better. 
Over half of the owners on low bid contracts rated the administrative effort as higher 
than normal. 



Owner Administrative Effort 

Summary of Responses 




Figure 11: Owner Administrative Effort Summary 




In Figure 12, the contractors’ responses on their level of administrative effort are 
summarized. In all three types of contracts, over half the contractors rated the amount of 
administrative effort as higher than normal. This indicates that the contractors perceive 
very little difference in their administrative workload based on the contract format. 
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Contractor Administrative Effort 

Summaiy of Responses 
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Figure 12: Contractor Administrative Effort Summary 



Reasonableness of Owner’s Inspection and Quality Assurance Programs 



Reasonableness of Inspection 

Summary of Responses 
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Figure 13: Reasonableness of Inspection Summary 



P-RFP 



NP-RFP 

□ 

NP-LB 



Figure 1 3 shows a summary of responses from contractors who were asked to 
grade the reasonableness of the owner’s inspection and Quality Assurance programs. 
The contract format theoretically has little to do with the reasonableness of the owner’s 
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inspection and Quality Assurance efforts. After all, in JOC contracts with government 
employees conducting oversight, the main focus is on fair and accurate assessments of 
the contractors quality control program. It would be expected that in the majority of 
cases, the contractors would, at a minimum, view the owner’s efforts as usually fair and 
accurate, with an occasional inaccurate assessment. This definition fits the reasonable 
category in Figure 13 and the highest-frequency answers were at least reasonable, with 
Non-Partnered/Low Bid having a majority in the very reasonable category. 

However, in situations where trust is low, owner representatives tend to increase 
inspection efforts in an attempt to “keep the contractor honest.” Though sometimes the 
increase in monitoring is warranted, in other cases overzealous inspection may result, 
with a corresponding decrease in the quality of the contractor-owner relationship. It is 
interesting to note here that, although the Partnered responses outnumber the Non- 
Partnered responses 16 to 15, the number of Non-Partnered respondents who stated that 
the owner’s efforts were unreasonable or somewhat unreasonable outnumbered their 
Partnered counterparts bv a ratio of five to one. 

3.3 Relationships 

The maintenance of a good relationship between the owner and contractor is 
essential to the success of a JOC. The repetitive nature of job order negotiations, 
combined with the fairly open structure of the contract, means that parties that get along 
well in a team environment will do much better in a JOC than those who thrive on the 
well defined structure of the traditional competitively bid contract. For some, it is a 
welcome change from the adversarial roles inherent in lump sum contracting. For others 
who are resistant to change, the attempts to administer JOC projects with a competitive 
bid mindset tend to result in hard feelings that make collaboration and negotiation 
difficult. 

High levels of trust and communication, mutually beneficial negotiation, and 
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quick resolution of disputes are all indicators of a successful JOC contract, or any 
construction contract for that matter. Since Partnering was designed to strengthen 
performance in these areas, it is expected that higher scores will be particularly evident in 
the Partnered/RFP sample population. As for Non-Partnered/RFP contracts, it is 
expected that dispute resolution and ease of negotiations would be enhanced since the 
unit prices would allow the contractor a reasonable profit. 



Ease of Individual Job Order Negotiations 

The trends in Figure 14 are in line with expectations. The most difficult 
negotiations from both owner and contractor perspectives are in the Non-Partnered/Low 
Bid arena. Scores increase as low bid is replaced by RFP, then further increase with the 
addition of Partnering. Three interesting items are apparent from this summary. 



Ease of Job Order Negotiations 

Summary of Sample Means 




Sample Population 

Figure 14: Ease of Job Order Negotiations Summary 

First, the contractors across the board perceive negotiations are more difficult in 
JOC than the owners. This could be due to the fact that the contractors expend much 
more effort in a given negotiation than the owners do, as they prepare the proposal and 
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must justify the requirements for line items to the owner. This might also indicate that 
owners find JOC negotiations easier in general compared with their experience in more 
adversarial forms of contracting, while the contractors are basing their expectations on 
their similar experience with other JOC projects. 

> * 

Second, the contractor score trend from Non-Partnered/Low Bid to 
Partnered/RFP is approximately linear, suggesting that the contractors’ perceptions of 
negotiation are improved equally by the additions of source selection and Partnering. 

Third, the owner scores increase the most when RFP source selection replaces 
low bid procurement, then slightly increase with the addition of Partnering. This would 
indicate that the most significant factor to the owner’s perception of successful job order 
negotiations is the method of contractor selection. 



Resolution of Changes, Claims, and Disputes 

Both the owners and contractors were asked the same question on the resolution 
of changes, claims, and disputes that arose within their contract. The descriptive answers 
were amplified with definitions. An answer of exceptional meant that issues were quickly 
resolved with mutual benefit. Good meant most were quickly resolved with mutual 
benefit, though a few required lengthy negotiations. Fair was similar to good, except a 
few issues required more formal dispute resolution techniques, such as dispute review 
boards and mediation. Finally, an answer of poor indicated that the parties consistently 
required lengthy negotiation, formal dispute resolution, or litigation to resolve issues. 
Figures 15 and 16 reflect the owner and contractor perceptions of dispute resolution, 
respectively. 



Dispute Resolution-Owner View 

Summary of Sample Means 
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POOR FAIR GOOD EXC 
Answer Category 



P-RFP 



NP-RFP 

□ 

NP-LB 



Figure 15: Dispute Resolution Summary - Owner View 

The inherent nature of the JOC process encourages early dispute resolution, since 
the owner retains the option of issuing work under a JOC or through another contracting 
vehicle. If a JOC is plagued with unresolved disputes or pending claims, the owner is less 
likely to issue future delivery orders. Most large JOC contractors maintain a policy of 
using claims or litigation as a last resort in recognition of this fact, and unless significant 
sums of money are at stake, it is generally recognized that a loss on one or two delivery 
orders is preferable to 30 to 40 lost delivery orders. Therefore, the expectation was to 
see very few answers below fair, with most in the good category. 



Dispute Resolution-Contractor View 

Summary of Responses 




Answer Category 
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□ 
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Figure 16: Dispute Resolution Summary - Contractor View 
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The summary of responses in Figures 15 and 16 are in line with expectations. 
Over half of the respondents rated dispute resolution as good or better. Half the owners 
using Partnered/RFP contracts rated dispute resolution as excellent. The single poor 
response came from a Non-Partnered/RFP contractor. 



Level of Trust 

Both owners and contractors were posed the same question on the level of trust 
that exists among project participants as compared with experience in similar contracts, 
or their expectations if this was the first JOC contract at the site. The five descriptive 
answers ranged from very low to very high. 

An analysis of the summaries of responses for both owner and contractor in 
Figures 17 and 18 both indicate that the Partnered group has higher levels of trust than 
both the Non-Partnered groups. Within the Non-Partnered groups, owners perceived 
higher levels of trust in the RFP contracts as compared with the low bid contracts. 



Level of Trust 

Owner Perspective 




Answer Category 



Figure 17: Level of Trust - Owner Perspective Summary 
From the owner’s perspective, most responses for both RFP groups were higher 
than normal or very high, while over half in the Non-Partnered/Low Bid group rated 
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trust as average or low. 81% of Partnered/RFP owners felt their sites maintained a 
higher than normal or very high level of trust, compared with 63% of Non- 
Partnered/RFP sites and 33% of Non-Partnered/Low Bid sites. 



Level of Trust 

Contractor Perspective 
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Figure 18: Level of Trust - Contractor Perspective Summary 



From the contractor’s point of view, the highest frequency answers were the 
same as the seen in the owners’ responses. 75% of Partnered/RFP contractors perceived 
a higher than normal or very high level of trust, compared with 44% of Non- 
Partnered/RFP sites and 33% of Non-Partnered/Low Bid sites. 

Comments from many respondents emphasized the importance of trust in the 
success of a JOC contract. In particular, contractors from Partnered/RFP sites were the 
most vocal in its promotion and made the following comments: 

- “Continuous building of TRUST between contractor/owner/customer is crucial. 
Give & take is necessary to make the contract work.” 

- “The University is an honest and straightforward owner. It is these two 
qualities that make this a workable relationship. If they were not open and trustworthy, 
we would not be able to respond to their need as promptly, effectively, and efficiently.” 
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- “High levels of trust and great partnership help to accomplish our common 

goals.” 

- “The reason that the JOC has been so successful here is because of the trust 
that has been developed between the contractor, Corps of Engineers, and the owner.” 

Trust is valued from Non-Partnered sites as well. One Non-Partnered/RFP 
contractor replied, “Although there is no formal Partnering process utilized on this 
project, there is a strong ‘Team’ working relationship with our government counterparts. 
Building and maintaining trust is the key to the success of a JOC.” 



Level of Communication 

As with level of trust, the level of communication was measured from both the 
contractor and owner perspectives. Again, answers ranged from very low to very high. 
Figures 1 9 and 20 summarize the responses, which both demonstrate that the Partnered 
group has higher levels of communication than both the Non-Partnered groups. Within 
the Non-Partnered groups, owners perceived higher levels of communication in the RFP 
contracts when evaluated against the low bid contracts. An impressive 94% of 



Level of Communication 




Answer Category 



P-RFP 



NP-RFP 



□ 

NP-LB 



Figure 19: Level of Communication - Owner’s Perspective Summary 
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Partnered/RFP owners felt their sites maintained a higher than normal or very high level 
of communication, compared with 50% of Non-Partnered/RFP sites and 33% of Non- 
Partnered/Low Bid sites. 75% of Partnered/RF P contractors perceived a higher than 
normal or very high level of communication, compared with 56% of Non-Partnered/RFP 
sites and 33% of Non-Partnered/Low Bid sites. 

Level of Communication 

Contractor Perspective 



P-RFP 
NP?RFP 

n 

NP-LB 
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Answer Category 

Figure 20: Level of Communication - Contractor Perspective Summary 

This comment from a Non-Partnered/RFP owner underscores the importance of 
communication: “JOC requires constant communication and close monitoring since we 
lack a formal set of drawings and specifications. The technical specs that are 
incorporated into the contract are so generic they have little impact. Questions regarding 
materials, methods, scheduling, field adjustments, and customer-generated requests for 
changes must constantly be addressed.” 
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3.4 Measurement of Overall Satisfaction 



Though much data has been studied on individual performance factors of the 
JOC process, the most significant measurement of contract success is taken from the 
standpoint of overall satisfaction with the project. A symphony may have many talented 
musicians who play flawlessly, but unless all the instruments are playing well and the 
conductor orchestrates them properly, the audience simply will not enjoy it. 

In the case of a construction contract, the audience per se is the end-user of the 
JOC’s construction services and the one that will live with the results long after the 
contractor demobilizes. In addition to measuring the end-users satisfaction, the 
contractor and owner also must provide their opinion on how successful the contract has 
been. Theoretically, Partnering and source selection should enhance the satisfaction 
levels of both the contractor, since disputes will be reduced in number and resolved 
quicker, hopefully leading to a mutually beneficial result. 

The efficiency of JOC will be compared with other methods to see if Partnering 
and source selection enhance the efficiency of the JOC process. 




Customer (End-user) Satisfaction 

The sample means displayed in Figure 21 show that the Non-Partnered/RFP 
scores are superior to Non-Partnered/Low Bid, with the Partnered/RFP mean much 
higher than the two other populations. The trend is approximately linear. The clear 
implication is that the Partnered/RFP format results in the highest levels of customer 
satisfaction, though in a Non-Partnered environment, RFP is superior to Low Bid. 
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Customer (End-user) Satisfaction 

Summary of Sample Means 




P-RFP NP-RFP NP-LB 



Sample Population 

Figure 21: Customer Satisfaction Summary 



Owner and Contractor Overall Satisfaction 

The owner responses presented in Figure 22 clearly indicate much higher levels 
of satisfaction with Partnered/RFP contracts. More significantly, however, 100% of 
Partnered/RFP owners surveyed report Higher-than-normal or Very-high levels of 
satisfaction, compared with 63% of Non-Partnered/RFP owners and 33% of Non- 
Partnered/Low Bid owners. 



Overall Level of Satisfaction with JOC 

Owner Perspective 
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□ 
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Figure 22: Overall Satisfaction Summary - Owner 




Overall Level of Satisfaction with JOC 

Contractor Perspective 
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Answer Category 
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Figure 23: Overall Satisfaction Summary - Contractor 



As expected, the contractor responses presented in Figure 23 parallel the owner 
answers. 94% of the contractors at Partnered/RFP sites reported higher than normal or 
very high levels of satisfaction, compared with 44% and 33% in the Non-Partnered/RFP 
and Non-Partnered/Low Bid populations, respectively. Once again, the Partnering/RFP 
sites display much higher levels of overall satisfaction with the contract. 



Efficiency of JOC contracts 



Efficiency of JOC vs. Other Methods 

Owner Perspective 








Figure 24: JOC Efficiency Summary 
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Since the JOC process is inherently more efficient for the completion of small 
projects than traditional sealed bid methods and other types of contracting, the data in 
Figure 24 was expected to slant toward the “better than” side of the scale. 69% of 
Partnered/RFP sites were rated much better than other contracting methods as compared 
with 50% for Non-Partnered/RFP and 33% for Non-Partnered/Low Bid. 



Comments from Respondents ‘ 

Though comments made at the end of the questionnaire were not requested for 
specific comparison, the respondents expressed some valuable information that 
illuminated the response received. Where possible, comments that were germane to an 
area under analysis were presented along with the data. 

However, one area of interest discussed more than once in the comments that 
was not part of the study concerned the number and stability of the personnel involved in 
the JOC process. Respondents specifically identified this area as having a major impact 
on the success of the project. The following comments were made on the subject of 
staffing and administration: 

- Contractor: “We have been here for 8 years working with the same people day 
in and day out and we have become a team.” 

- Contractor: “We have a very good working relationship with the Army and 
COE. I attribute our continued success (2 JOC projects) to stability of the [contractor] 
team, our knowledge of construction, and our close relationships with the owner.” 

- Contractor: “The process of construction demands consistency in expectation 
by the contractor from the owner (gov). Probably the greatest deterrent to a successful 
Partnering is how each SABER contracting officer interprets their duty requirements of 
the contractor. With each new change of personnel, a new way of doing things is 
implemented, some better than others but a moving target at best.” 
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- Contractor: “We have had to work hard at this contract, probably because of 
the number of people involved. We are working with 4 Corps of Engineer districts in 5 
different states. Negotiations are held with quite a large number of participants. 
Inspections, as you can imagine, can be trying.” 

- Owner: “The JOC process is a good one if the contractor takes a serious 
interest in its success. We have been exposed to about four project managers during the 
9 months of this contract and at times we were unsure of who was truly in charge.” 

- Contractor: “[Owner] has the most frequent turnover of both contracting & 

CE staff and the vast majority are trainees.” 

Clearly, both owner and contractor should attempt to keep turnover to a 
minimum in JOC and aim for consistency in contractual expectations by keeping the 
number of personnel down to a manageable level. 

3.5 Statistical Analysis 

The data collected through the survey responses was subjected to analysis to 
determine its statistical significance. A numerical score was first assigned to every 
answer. For those questions that had already been reported as numbers, they were scored 
as reported. For descriptive answers, numbers were assigned to each category according 
to the following convention: 

- For answers ranging from “Very Low” to “Very High”: 1 - 5 

- For answers ranging from “Poor” to Exceptional: 1 - 4 

- For answers ranging from “Much Worse Than” to Much Better Than”: 1 - 5 

Next, a series of histograms was developed to determine if a normal distribution 
of data existed for answers given by a certain sample population. If two sets of data to be 
compared were of a normal distribution, a comparison of small sample means via the t- 
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test was used. If the data did not meet that criteria, the Mann- Whitney U-Test of sample 
medians was employed. 

In comparing the data, the response scores for each question was analyzed in 
three separate tests: 

1. Non-Partnered/RFP vs. Non-Partnered/Low Bid 

2. Non-Partnered/RFP vs. Partnered/RFP 

3. Partnered/RFP vs. Non-Partnered/Low Bid 

This resulted in 84 comparisons (28 questions x 3 tests). Of all the comparisons 
conducted, only three could be performed by the small sample means test, with the rest 
conducted by the Mann- Whitney U-Test. It was decided to run all 84 comparisons 
through the Mann-Whitney U-Test as well as perform the three small sample means 
tests. All tests were conducted with a significance level of 0.05. 

The development of a decision rule centered around a one-tail test. Three 
different decision rules were developed according to the expectations of the responses: 



Decision Rule #1 - Non-Partnered/RFP vs. Non-Partnered/Low Bid 

Null Hypothesis: Performance of Non-Partnered/RFP projects is the same as or inferior 
to Non-Partnered/Low Bid projects. 

Alternate Hypothesis: Performance of Non-Partnered/RFP projects is superior to Non- 
Partnered/Low Bid projects. 

Decision Rule #2 - Non-Partnered/RFP vs. Partnered/RFP 

Null Hypothesis: Performance of Partnered/RFP projects is the same as or inferior to 
Non-Partnered/RFP projects. 

Alternate Hypothesis: Performance of Partnered/RFP projects is superior to Non- 
Partnered/RFP projects. 
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Decision Rule #3 - Partnered/RFP vs. Non-Partnered/Low Bid 

Null Hypothesis: Performance of Partnered/RFP projects is the same as or inferior to 
Non-Partnered/Low Bid projects. 

Alternate Hypothesis: Performance of Partnered/RFP projects is superior to Non- 
Partnered/Low Bid projects. 



The three small sample means tests did not produce statistically significant 
results. Of the 84 Mann- Whitney U-Test comparisons conducted, there were only 13 
cases where the data was statistically significant and the null hypothesis was rejected. 
These cases are presented in the form of the alternate hypothesis: 



Non-Partnered/RFP vs. Partnered/RFP 

1. Owner Question # 3 : The quality and frequency of innovative ideas and Value 
Engineering proposals from contractors within Partnered/RFP contracts are superior to 
Non-Partnered/RFP contracts. 



2. Owner Question # 4: Contractor responsiveness to client needs within Partnered/RFP 
contracts is superior to Non-Partnered/RFP contracts. 



3. Contractor Question # 7: The level of trust within Partnered/RFP contracts is superior 
to Non-Partnered/RFP contracts. 



4. Contractor Question U 9: The overall satisfaction of contractors within Partnered/RFP 
contracts is superior to Non-Partnered/RFP contracts. 



) 
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Partnered/RFP vs. Non-Partnered/Low Bid 

5. Owner Question #1: The responsiveness and timeliness of cost estimates and “basic” 
drawings from contractors within Partnered/RFP contracts is superior to Non- 
Partnered/Low Bid contracts. 



6. Owner Question #8: The contractors’ ability to prevent and solve scheduling and site 
coordination problems within Partnered/RFP contracts is superior to Non- 
Partnered/Low Bid contracts. 



7. Owner Question #10: Customer (end-user) satisfaction within Partnered/RFP 
contracts is superior to Non-Partnered/Low Bid contracts. 



8. Owner Question #11: The management effectiveness of contractors within 
Partnered/RFP contracts is superior to Non-Partnered/Low Bid contracts. 



9. Owner Question #17: The required level of administrative effort within Partnered/RFP 
contracts is less than within Non-Partnered/Low Bid contracts. 



10. Owner Question #18: The level of trust within Partnered/RFP contracts is superior 
to Non-Partnered/Low Bid contracts. 



11. Owner Question #19: The level of communication within Partnered/RFP contracts is 
superior to Non-Partnered/Low Bid contracts. 
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12. Owner Question #20: The owners’ level of overall satisfaction within Partnered/RFP 
contracts is superior to Non-Partnered/Low Bid contracts. 



13. Contractor Question #9: The contractors’ level of overall satisfaction within 
Partnered/RFP contracts is superior to Non-Partnered/Low Bid contracts. 
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Chapter IV: Conclusions and Recommendations 



Conclusions 

Job Order Contracting is a very popular method with facility owners who need a 
responsive method for accomplishing small construction and repair projects in a 
streamlined manner. However, the relationships of the parties in such an open and 
flexible structure plays a much greater role in the project’s success as compared to more 
traditional contracting methods. The formal Partnering process that serves to strengthen 
and focus these relationships toward a common vision of success should therefore have a 
significant impact on the performance of JOC contracts. In addition, the selection of a 
qualified, responsive JOC contractor through a source selection process should yield 
better performance then can be gained through competitive bid. 

Though this study did not produce a large volume of statistically significant data 
with which to draw sweeping inferences, a few key areas studied did show that owners 
and contractors feel that Partnering and RFP source selection add value to the JOC 
process. Among projects that were procured through RFP source selection. Partnered 
contracts had significantly higher levels of contractor innovation, contractor 
responsiveness to client needs, trust among project participants, and contractor 
satisfaction. 

When Partnered contracts procured through RFP source selection were 
compared with Non-Partnered, low competitive bid contracts, the results were more 
impressive. From the owner’s perspective, the contractors in the Partnered/RFP format 
were much more responsive with cost estimates and required drawings, displayed greater 
management effectiveness, and were more able to solve and prevent scheduling and site 
coordination problems. In addition, Partnered/RFP owners reported a reduced level of 
administrative efforts in their projects. Among project participants, those in the 
Partnered/RFP population enjoyed higher levels of trust and communication than their 
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Non-Partnered/Low Bid peers. 

The most significant result in the comparison of Partnered/RFP projects to Non- 
Partnered/Low Bid projects was in the area of overall satisfaction. A consensus of the 
three different groups involved in the process, owner, contractor, and end-user, all 
reported much higher levels of satisfaction under the Partnered/RFP format. Since this 
takes into account how all the various elements of the contract perform as a whole, it is 
very strong endorsement of Partnering and source selection. 

The results can be summed up in an especially poignant comment from an owner 
in the Partnered/RFP group: 

- “I have been involved with JOC going on 1 1 years. The lesson that I have 
learned is to have a good relationship with the contractor and have a contractor with a 
good track record... .Do some calling, check other places where the contractor has 
worked, and ask about their performance. This could save a lot of problems later on.” 



Recommen dations 

Based on the study results, facility managers who want to have a responsive and 
cost-effective contract that is relatively easy to administer and generates high levels of 
customer satisfaction need to select JOC contractors through a process of source 
selection that values past performance and qualifications. Artificially low coefficients 
should be avoided. Once a contractor is selected, the owner and contractor should 
implement a robust Partnering program that is continually championed and reinforced. 
Personnel changes should be kept to a minimum, but new personnel should be 
indoctrinated as to the common goals in the Partnering charter as they join the team. 

Many of the Mann- Whitney U-Tests that were run and did not produce 

* 

significant statistical data were very close. In fact, 24 of the 71 failed tests (34%) scored 
90% or more of the points required to prove statistical significance. This is attributed to 
the small sample size in the Non-Partnered populations. A further study could expand the 
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number of sites included in the survey and possibly produce more statistically significant 
results. 

The primary basis for this study was subjective opinions of contractor 
performance. Another study might include objective measures of performance such as 
cost escalation, on-time completion of delivery orders, and other similar metrics. 
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